People and Cookies: Imperfect Treatment Assignment in Online Experiments

Identifying the same internet user across devices or over time is often infeasible. This presents a problem for online experiments, as it precludes person-level randomization. Randomization must instead be done using imperfect proxies for people, like cookies, email addresses, or device identifiers. Users may be partially treated and partially untreated as some of their cookies are assigned to the test group and some to the control group, complicating statistical inference. We show that the estimated treatment e↵ect in a cookie-level experiment converges to a weighted average of the marginal e↵ects of treating more of a user’s cookies. If the marginal e↵ects of cookie treatment exposure are positive and constant, it underestimates the true person-level e↵ect by a factor equal to the number of cookies per person. Using two separate datasets—cookie assignment data from Atlas and advertising exposure and purchase data from Facebook—we empirically quantify the di↵erences between cookie and personlevel advertising e↵ectiveness experiments. The e↵ects are substantial: cookie tests underestimate the true person-level e↵ects by a factor of about three, and require two to three times the number of people to achieve the same power as a test with perfect treatment assignment.